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Abstract 

The National Association of Boards of Pharmacy (NABP) manag <>g the 
National Association of Boards of Pharmacy Licensure Examination (NABPLEX). 
A new Scope of Pharmacy Practice Study was completed in 1994, and representatives 
of NABP and the National Association of Pharmacy Regulatory Authorities 
(NAPRA) of Canada met in 1995 to discuss the possibility of creating an 
international pharmacy licensing examination. The first step was to determine 
whether a common examination blueprint between the two countries was feasible. 
The NABPLEX competency statements were updated based on the results of the 1994 
study, and they were then reviewed independently by representatives of each country 
to assess their applicability to each coimtry. A survey instrument was developed to 
assess the relative importance of each competency statement as it relates to entry-level 
pharmacy practice. Results were analyzed separately for American respondents, 
Canadian respondents, and both groups combined, and a joint examination blueprint 
was approved by both groups. The focus of this paper is on the process used to 
establish the joint blueprint, rather than on the blueprint itself. 

Introduction 

The NABPLEX is a licensing examination designed to protect the health and welfare 
of the public by assessing the competence of candidates to function safely as entry-level 
pharmacists. In order to allow valid inferences to be made about candidate performance on 
the job on the basis of their performance on the examination, however, it is critical that the 
examination focus on the tasks that are required of entry-level practitioners, and that the 
most important job activities carry the greatest weight in the examination. Much has been 
written, and continues to be written, about the importance of conducting ftmpir iral studies 
as a basis for assigning weights to the topics covered in a credentialing examination. 
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standard 11.1 of the Standards for Educational and Psychological Testing (American 
Educational Research Association, American Psychological Association, and National 
Council on Measurement in Education, 1985), for example, specifies that: 

"The content domain to be covered by a licensure or certification test should be 
defined clearly and explained in terms of the importance of the content for competent 
performance in an occupation. A rationale should be provided to support a rlaitn 
that the knowledge or skills being assessed are required for competent performance 
m an occupation... " (p. 64). 

The comment related to that standard specifies that "job analyses provide the primary basis 
for defining the content domain" (p. 64). Procedures for establishing weights on the basis 
of job analysis data, or "practice analysis" data as it has been described more recently (Kane, 
1997), have been described in a number of articles. Kane, Kingsbury, Colton, «& Estes 
(1989), provided a thorough explanation of the importance of controlling the relative 
contributions of criticality and frequency ratings m establishing estimates of importance, and 
Lunz, Stahl, & James (1989) provided a concise desaription of how a Rasch rating scale 
analysis could be applied to translate the results of a job analysis into test specifications. 
The issues addressed in each of these articles were considered in depth in devising the 
analyses for the present study. 

A study was completed in 1994 to delineate the current job or practice-related 
activities of pharmacists. It was sponsored jointly by the American Association of Colleges 
of Pharmacy (AACP), American Pharmaceutical Association (APhA), American Society of 
Hospital Pharmacists (ASHP), and National Association of Boards of Pharmacy (NABP). 
The 1994 study did not, however, focus specifically on entry-level pharmacy practice, nor 
did it address the "importance" of practice-related activities in a way that led to development 
of a test blueprint. Hence the need for the present study. 

The present study grew out of the need to review the NABPLEX blueprint in light 
of the results of the 1994 study, and to update it if necessary. In addition, there had been 
discussions between representatives of the United States and Canada regarding the feasibility 
of developing a single Kcensure examination that would meet the needs of both countries for 
assessing the competrace of entry-level pharmacists. Consistent with the goals of the North 
American Free Trade Agreement (NAFTA), creation of such an examination would help to 
remove the bamers associated with allowing qualified pharmacists to work in either country. 

The purpose of the present study was first to assess the feasibility of establishing a 
common blueprint for an international pharmacy licensure examination that would be 
appropriate for use in both the United States and Canada, arid if appropriate, to document 
the process by which such a blueprint could be established. 
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Methodology 



As used here, the term "test blueprint" includes both the topics to be covered by an 
examination, and the weights assigned to each topic. As a result, the feasibihty of a 
common blueprint was assessed in two phases. Phase 1 involved creating competency 
statements that content experts from both countries agreed delineated those aspects of 
pharmacy practice that have an impact on protecting the health and welfare of the public. 
Phase 2 involved assessing the relative importance of each competency statement, and 
determining whether the weights that would be assigned to each would be the same for both 
countries. 

Developing Compe tency Statements 

The existing NABPLEX competency statements were reviewed in light of the results 
of the 1994 study, and revised wherever necessary to ensure that they would continue to 
delineate contemporary entry-level pharmacy practice. The revised competencies were 
reviewed independently by the NABPLEX Review Committee (NRC), the Advisory 
Committee on Examinations (ACE), representatives of NAPRA, and NABP staff. 
Representatives of ACE and NAPRA then came together to resolve any discrepancies tha t 
arose from the separate deliberations, and to discuss whether the competency statements 
adequately delineated entry-level pharmacy practice in both coimtries. 

Based on discussion during the meeting, it was clear that all of the competency 
statements applied to both coimtries, and neither group anticipated that additional 
competency statements would be required in order adequately delineate entry-level pharmacy 
practice. From the perspective of topics to be included in the blueprint, then, a single 
blueprint for both countries was seen as definitely feasible. Based on that, the second phase 
of the feasibility study was undertaken, determining whether a single set of weights for the 
competency statements would apply to both countries. 

Ass ig ni ng Weights to the Competency Statements 

In order to gather data rega r d in g the relative importance of the various competency 
and subcompetency statements that delineated entry-level pharmacy practice, the revised 
competencies were transformed into a survey instrument. A hierarchical structiue was 
established by subdividing pharmacy practice into three major functional areas, and further 
subdividing those areas into competency statements and subcompetency statements. There 
were a total of nine competency statements, and thirty-nine subcompetency statements. A 
complete list of these is included as Appendix A. A review of the competencies within the 
survey instrument shows that the three major competency areas were very broad atiH general, 
the nine competency statements were more specific, but stiU fairly broad, and the 39 
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subcompetency statements were more specific yet. 

An article by Kane, Kingsbury, Colton, & Estes (1989) provided practical guidelines 
for assessing the relative importance of practice-related activities. 

"The relative importance of any activity in practice will depend on the frequency of 
the activity (how often it is performed) and the criticality of the activity (the 
difference that it makes in terms of client outcomes). The results of the job analysis 
can be summarized in terms of the average frequency of occurrence of each activity 
over respondents and the average rating of criticahty over respondents. The central 
task is then to combine average frequency and average criticahty in order to get an 
overall index of the importance of the activity." (p. 19) 

This is consistent with the expectation that, in order to be considered competent, a 
practitioner should make very few if any serious errors, and few errors of any kind that have 
an impact on the health and welfare of the pubhc. From that perspective, ratings of 
frequency and criticahty are both indicators of the overall importance of an activity. 

In order to assess the relative importance of the activities delineated in the survey 
then, five-point rating scales were established to assess the criticahty (i.e., seriousness of 
consequences) of each delineated activity, and to assess how frequently an entry-level 
pharmacist would be likely to perform the activity. In addition, the survey included a 
section on the demographic and professional characteristics of the respondents, and a section 
for quahtative comments about the survey. Samples of the rating scales and related 
instructions are included below: 



Please use the following Criticality and Frequency rating scales in all of your ratings: 



Criticality 


Frequency 


Generally, how serious are the consequences 
(e.g,, harm to patient) if the competency is not 
performed properly? 


How often does an entrydevel 
pharmacist in your practice setting 
perform the competency? 


1 Not serious (eg., has no effect) 

2 Minimally serious (e.g., causes inconvenience) 

3 Moderately serious (eg., hinders therapeutic 
progress) 

4 Highly serious (e.g., worsens the padenfs 
condition) 

5 Critically serious (eg., is life threatening) 


1 Very rarely (eg., monthly or less) 

2 Rarely (eg., weekly) 

3 Occasionally (eg., daily) 

4 Often (e.g., hourly) 

3 Very often (eg., many times per hour) 



For example, consider the competency "Evaluate drug therapy for the presence of phaimacotherapeudc 
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duplications and interactions. " Although it is true that some drug duplications and/or interactions maybe 
without clinical impact while others may be literally life threatening please form your judgment by integrating 
the range of possibilities - if, overall, you believe that, generally, therapeutic progress will be hindered if drug 
duplications and/or interactions are not evaluated and identified, circle the "3" in the column labeled 
“Criticality. “ If in your practice setting entry-level pharmacists evaluate drug therapy for the presence of 
duplication/interactions often (e.g. hourly), circle the “4“ in the column labeled “Frequency. “ The table below 
illustrates bow to mark your desired ratings. 



Competency 


Criticality 

Rating 


Frequency 

Rating 


Evaluate drug therapy for the presence of 
pbarmacotherapeutic duplications and interactions 


1 2 04 5 


1 2 3^5 



You may choose to read the competencies and make both your criticality and your frequency ratings at the 
same time, or you may choose to first rate the criticahty of all of the competencies, and thi‘n go through the 
competencies making your frequency ratings. 



After the survey instrument had been finalized it was pilot tested on a CTnall group 
of licensed p h a rma cists to ensure that the instructions were clear and complete, and the 
rating scales were workable for delineating the competency areas of interest. Representatives 
of NAPRA produced French and Hnglish versions of the survey that were appropriate for 
distribution in Canada, taking into account differences between the spelling and language 
conventions of the two countries. Based on the results of the pilot test, no changes to the 
survey instrument were required. The number of French sm^eys completed and retiuned 
was, however, very small and insufficient to support meaningful separate analyses of the 
ratings. 

An important feature of the survey instrument was the fact that rating scales were 
attached to both the competency statements and the subcompetency statements. This 
permitted two ways of generating weights for the competencies. Weights for the nine 
competency statements could be generated directly on the basis of the ratings of frequency 
and criticality for those statements. Alternatively, weights could be generated first for the 
39 subcompetency statements based on their ratings of frequency and criticality, and those 
weights could then be combined by summing them appropriately to obtain weights for the 
nine competency statements. Because of the hierarchical nature of the survey instrument, 
working from the more general statements to the more specific, i.e., from the top of the 
hierarchical structure down, the analyses that began with ratings of competency statements 
are referred to as "top down" analyses, and those that began with ratings of the 
subcompetency statements are referred to as "bottom up" analyses. In a true top down 
analysis, rating scales could also have been attached to the three major competency areas. 
These were, however, judged to be too general to be amenable to ratings of frequency and 
criticality in the present study. 
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D istributing and Retrieving the. Siirvf»v<i 



In the United States, surveys were distributed to a geographically representative 
random sample of 5,600 licensed pharmacists, which was drawn from the Pharmacy 
Manpower Database of 126,286 records. Thus the sample included approximately 4.4% of 
the licensed pharmacists in the database. In addition, the drafted competency statements 
were distributed to all state boards of pharmacy for comment, and a press release, 
accompanied by a copy of the survey, was distributed to all boards of pharmacy, schools 
and colleges of pharmacy, state and national pharmacy associations, and the pharmacy 
press. 



In Canada, surveys were distributed to a geographically representative sample of 
licensed p h a rm acists, based on information provided by NAPRA. The total number of 
licensed pharmacists in each province or territory was estimated, and a random sample of 
10% of the licensed practitioners in each area was targeted to receive the survey, with a 
mi nim u m of 40 per jurisdiction wherever possible. NAPRA assumed responsibility for 
distributing and retrieving the surveys in Canada, and forwarded the completed surveys to 
NABP for processing. The total number of Canadian surveys sent was 1,738. The larger 
sampling percentage in Canada compared to the United States was intentional, in part to 
help ensure that sufficient completed surveys would be available to support the analyses 
based on Canadian respondents only, and in part to ensure that the ranaHiim responses 
would not be overwhelmed when they were combined with the American responses. 



In order to improve response rates, a letter was sent from the Executive Directors of 
NABP and NAPRA in advance of the svurvey, to all potential participants. The purpose of 
the letter was to explain the importance of the survey and to encourage participation. A 
follow-up postcard was also sent two weeks before the surveys were due, encouraging people 
who had not retximed the completed surveys to do so. Postage-paid return envelopes were 
included with the surveys, to simplify the respondents’ task in returning them. 

Analyzin g the Completed Surveys 

The purpose of the survey was to assess the relative importance of the competency 
and subcompetency statements. Although the testing standards provide some general 
guidelines regarding how to proceed toward the development of a blueprint, much remains 
to the discretion of the committees and researchers. Should frequency carry the weight 

as criticahty, for example, in assessing the relative importance of practice-related activities? 
Should primary consideration be given to the importance of the competency statements, or 
to the importance of the subcompetency statements, and does it malfp a difference? In the 
absence of clear philosophical answers to questions such as these, it was decided that the 
best overall approach would be to assess their empirical consequences, and talw those 
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consequences into account in approving the final blueprint. As a result, alternative 
blueprints were generated on the basis of various combinations of assumptions, and 
presented for consideration by the panels of content experts. Their task was to decide 
whether one solution made more sense than the others from a practitioner perspective. 

A total of 1,019 practicing pharmacists in the United States, and 358 in Canada 
responded to the survey. Taking into account surveys that were non-deliverable due to 
changes in address, and those in the samples who responded that they were not actively 
practicing pharmacy, this corresponded to response rates of approximately 25% in the 
United States, and 21% in Canada. The relatively low overall response rates were 
disappointing, given the steps that had been taken to encourage people to complete and 
return the surveys. A comparison of the demographic characteristics of the American 
respondents to the corresponding characteristics of the Manpower Database showed the 
respondents to be very representative. Similar comparisons could not be made for the 
Canadian sample, because a comparable Canadian database was not available. A review 
of the demographic characteristics of both the American and Canadian samples did, 
however, show that pharmacists having a wide range of experience, representing a wide 
range of practice settings did complete the survey. Completed surveys were received from 
practitioners in all fifty states and all ten provinces. 

One indication that the overall results obtained were quite stable, despite the CTnall 
number of Canadian respondents, was the fact that the preliminary results were generated 
for Canadian respondents immediately after the deadline for returning the surveys, based on 
223 respondents, and again before the joint panel review meeting, based on 280 P.anadian 
respondents. The results changed only fractionally based on the increased sample size. An 
additional 78 Canadian surveys were returned after the deadline for preparing materials for 
review by the panels of content experts, so they were not included in the preliminary 
analyses that were discussed during the panel review. They were later incorporated into the 
data set and all of the statistical analyses were rerun. The updated results were discussed 
with the panel via conference call, but again, incorporating the additional responses into the 
data set changed the results only fractionally. Adding the responses did not result in a 
different number of questions being assigned to any competency area, based on the 
assumption of a 150-item examination. Based on tha t, it appeared that incorporating more 
completed surveys into the data set would have changed the results very httle if at all. 

The last section of the survey provided respondents with an opportunity to 
recommend additional competencies, in case significant competency areas had inadvertently 
been omitted. If any such topics had been identified, a follow-up survey would have been 
conducted to establish weights for those topics. Based on a review of the comments 
received, however, no additional topics were required. 
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A Rasch rating scale analysis using the computer program Bigsteps (Wright, B.D., 
and Linacre, J.M., 1992) was run to transform the ratings of criticality and frequency 
associated with each of the nine competency statements onto a single linear scale. The 
transformed ratings were then combined as weighted averages to obtain "importance" 
estimates for each competency statement. As pointed out by Kane, Kingsbury, Colton, & 
Estes (1989), 

"The relative emphasis that should be given to frequency and criticality in 
determining importance is a matter of judgment. However, for hcensure that is 
intended to protect the pubhc from harm or unnecessary risk, criticality would seem 
to be of at least as much concern as frequency." (p. 20) 

A number of different importance estimates were generated for each statement by varying 
the relative contributions of criticality and frequency to the importance variable. 
Specifically, ratios of criticality to frequency of O.T, l.T, 2:1, 4.T, and 1:0 were used to 
generate results. The O.T ratio, no weight given to criticality, and the 1:0 ratio, all weight 
given to criticality were considered only to describe the limiting extremes for the weights. 
Similarly, the l.T ratio, equal weight to criticality and frequency, was only considered as 
providing baseline data. The panel of content experts decided unanimously that, for 
purposes of hcensme, criticality should carry more weight than frequency in assessing the 
importance of a competency area. 

A second Rasch rating scale analysis was run on the cnticahty and frequency ratings 
for the 39 subcompetency statements, and as for the nine competency statements, the 
transformed ratings were then combined as weighted averages to obtain "importance" 
estimates for each subcompetency statement. As before, results were generated for ratios of 
criticality to frequency of O.T, l.T, 2:1, 4:1, and 1:0. 

In order to transform the estimates of importance into actual numbers of questions 
in the examination, or percentages of the total test, the following system of inequalities was 
set up and several linear programming analyses were run to generate solutions to the 
following constraints: 

■ The total number of questions in the examination was to be 150. 

■ The number of questions or percentage of the examination assigned to a 
competency stat^ent was to be greater than or equal to zero, and 
proportional to its estimated importance; the more important the statement, 
the greater the weight assigned to it. 

■ The number of questions or percentage of the examination assigned to a 
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subcompetency statement was to be greater than or equal to zero, and 
proportional to its estimated importance. 

The linear programming analyses distributed the 150 questions across the blueprint subject 
to these constraints, and the results are reported in the following section as percentages of 
the examination. As explained in the following sections, separate analyses were run for the 
competency statements and the subcompetency statements. 

Top Down Versu s Bottom Up De l ineations of Pharmacy Pranticft 

The fact that rating scales had been attached to both competency statements anH 
subcompetency statements in the survey made it possible to establish weights a^yy^jaf^ with 
pharmacy practice from two different perspectives, a "top down" view, and a "bottom up" 
view. To the extent that the overall results of the two approaches differed, this would 
provide the panel of content experts with two delineations of pharmacy practice, giving the m 
two options for the final blueprint. One reason to thiuk that the two approaches might 
produce different results was the fact that, because the competency statements ten ded to be 
more broad in scope than the subcompetency statements, they may have been more difficult 
to rate in terms of frequency and criticality. Another was the fact that the 39 
subcompetency statements were not distributed equally across the nine competency 
statements. Competency statement 1.2, for example, included seven subcompetencies, 
whereas competency statement 3.3 included only two subcompetencies. The bottom up 
analyses would likely have resulted in a relatively large weight being assigned to a 
competency such as 1.2 simply because that weight was obtained by adding seven smaller 
weights. Conversely, the top down analysis may have resulted in a relatively low weight 
being assigned to a competency statement such as 1.2 since it did not take into account the 
fact that competency 1.2 covered so many subcompetencies. Because there was no single 
"correct" way to analyze the data, results were generated on the basis of both the top down 
and bottom up analyses, and presented to panels of content experts for review anH 
consideration. 

With the top down approach, pharmacy practice was divided into nine parts, based 
on the importance estimates associated with each of the nine competency statements. The 
more important the competency statement, the larger the part. Each of those parts was then 
subdivided into smaller parts based on the importance estimates for the subcompetency 
statements that comprised the particular competency statement. 

With the bottom up approach, pharmacy practice was divided into 39 parts, based 
on the importance es tim a t es associated with each of the thirty-nine subcompetency 
statements. The weights assigned to each competency statement were then obtained by 
summing the weights associated with the subcompetency statements that belonged to the 
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particular competency statement. 



Complete analyses were run separately based on American respondents only, 
Canadian respondents only, and both groups combined. These included, for each group, 
running Rasch rating scale analyses to transform the ratings of criticahty and frequency onto 
a single linear scale; generating importance estimates based on weighted averages of the 
transformed ratings; and generating potential examination blueprints based on top down and 
bottom up analyses. 

Results 

Results summarized in the following Tables were discussed independently with panels 
of content experts from the United States and Canada, and then jointly with the two groups. 
The pxirpose of the discussions was to determine whether any of the analyses produced a 
workable picture of entry-level pharmacy practice, and if so, whether it would be possible 
to adopt one that would be appropriate for delineating entry-level pharmacy practice in both 
countries. Results of the top down analyses are summarized in Table 1. The weights 
assigned to each competency statement are reported as percentages of the total test. 

Table 1 

Results of Top Down Analysis 
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The resiilts of the top down analysis were not very encouraging for either group of 
reviewers. Neither group found any of the potential blueprints based on the top down 
approach to be believable representations of entry-level pharmacy practice. The most serious 
problem was the fact that competency area 1, which included competency statements 1.1, 
1.2, and 1.3, did not receive enough weight. Even though the purpose of the survey was to 
establish the appropriate weights, both groups of content experts believed that, based on its 
scope, content area 1 would have to be the most heavily weighted if the blueprint were to 
accurately represent entry-level pharmacy practice. 

Two findings that were noteworthy from the top down analyses were the fact that 
there was a good deal of similarity between the results obtained based on American 
respondents only and those based on Canadian respondents only, and the fact that giving 
additional weight to criticality compared to frequency had a small but noticeable effect on 
the final weights. This suggested that it was quite likely that a single blueprint could serve 
the needs of both countries, and that the decision regarding how much to weight criticality 
versus frequency ratings could be based on an assessment of the content-related implications 
of the various combinations. 

Results based on the bottom up analyses are summarized in Table 2. As explained 
earlier, these results were based on the analyses of ratings associated with the subcompetency 
statements. The weight associated with competency statement 1.1, for example, was 
obtained by summing the weights of the four subcompetency statements that comprise it. 
Results were presented in this way in part because they would then be directly comparable 
to those obtained from the top down analyses and in part because the final weights at the 
subcompetency statement level are considered confidential. The results of the bottom up 
analyses showed a marked difference from those based on the top down approach. 
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Table 2 

Results of Bottom Up Analysis 
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Criticality = 2 x Frequency 


CritkaHty^4xB»qiieacy 




j^ioeotage of Examtoadon 


Percentage of Examinatioa 




Competency 

Statement 


US Only 


Only; 


Combined 


US Only 


Only 


Combined 


US Only 


. Only. 


Oonihifted-, 


1.1 


9J2 


1 0,3 


9.4 


8.8 


9.8 


9.0 


8.5 


9,4 


• 


1.2 


24S 


24i . • 


' 252 


24.2 


24.7 


24.6 


23.5 


24.6 


24.0 


1.3 


lllllplll 


. * 


14.4 


14.6 


15.9 


15.0 


^15.0 




: .15.5/"’ 


2.1 


94 


.^1 


^7. 


10.5 


9.3 


102 


11.7 


103 > 




2.2 


1L2 


IM 


10.8 


10.5 


9.5 


10.0 


^.9 






2.3 


5.4 


sji 


’ -54 


5.9 


5.1 


5.6 


6.4 


* ’ \ ■ -S.! ^ 


4C0' 


3.1 


3.9 




3.6 


3.7 


2.9 


3.3 


3.5 


22 


3.0 


3.2 


17.5 


17.4 


17,8 


17.0 


17.6 


17.4 


16.6 


. 17.5 . 


17.0; 


3.3 


4.9 - 


• 5.2 


4.9 


4.9 


52 


4.9 


4.9 


' '32 •• 




Total* 


100 


100 


100 


100 


100 


100 


100 


100 


;ioo 



Totals may not add to 100.0% because of roimding. 



Both the American and Canadian content experts believed that the results of the 
bottom up analyses conveyed an accurate picture of entry-level pharmacy practice. Of 
particular importance to them was the reversal of weight assigned to competency areas 1 and 
2, compared to those based on the top down analysis. Both groups had believed very 
strongly that, from a content perspective, competency area 1 should have the greatest weight. 
That expectation was supported by the results of the bottom up analyses. Based on that, 
the answer was "yes" to the question of whether a suitable blueprint could be established on 
the basis of the survey results. 

Another si gnifi cant diflerenoe between the results based on the bottom up versus top 
down analyses was the fact that the bottom up analysis assigned some weight to competency 
statement 3.3, whereas the top down analysis did not. That competency statement, along 
with its two subcompetency statements, addressed a pharmacist’s responsibility for educating 
patients and the public regarding wellness, disease states, and medical conditions, and 
content experts from both countries believed very strongly that it would be unacceptable to 
eliminate it from the blueprint. 




Validation of the Bluepitot for an Intemational LioensiiTe F-TaTnirm tinn 



12 

BEST CX)PY AVAILABLE 



A comparison of the weights that were obtained on the basis of American 
respondents only and Canadian respondents only showed only very small differences, anH 
the weights obtained by combining both sets of responses were very representative of the 
separate ratings for each group. As noted for the top dovwa analyses, increasing the 
contribution of criticahty in comparison to frequency in estimating the importance of the 
competencies had a small but noticeable effect on the final weights that were obtained. The 
content experts from both countries agreed independently that even though the differences 
were s m a ll , the results based on cnticahty being weighted four times as heavily as frequency 
produced the best overall results. Based on that, the answer was again "yes" to the question 
of whether a single examination blueprint would meet the needs of both coxmtries. 

Working independently, and then together as a group, reviewers from both countries 
approved adopting the blueprint that was based on the bottom up analyses, for ratings based 
on both groups of respondents combined, with importance estimates based on criticahty 
ratings being weighted four times as heavily as frequency ratings. 

Summary and Discussion 

This paper described the process whereby a single examination blueprint was 
developed and adopted as appropriate for establishing a single hcensme examination to 
assess the competence of entry-level pharmacists in both the United States and Canada. 
Because the blueprint for such an examination would include both the competency 
statements according to which performance is to be assessed, and the weights assigned to 
those competency statements, the work described in this paper was completed in two phases. 

Phase 1 involved determining whether a single set of competency statements would 
apply to both coxmtries. If, for example, there were important competency areas that were 
important for practitioners in the United States, but not in Canada, or vice versa, a single 
blueprint would not have been viable. A set of competency statements delineating entry-level 
pharmacy practice was reviewed independently by panels of content experts from both 
countries to assess their applicabihty to each coimtry. Based on a joint discussion of the 
results of the independent reviews, the competency statements were judged to be appropriate 
for use in both the United States and Canada. 

As a safety precaution, the survey that .was developed to establish weights for the 
competency statements included a section in which respondents could identify additional 
topics that they felt should have been included. No significant additional competency areas 
were identified by either American or Canadian respondents. This led support to the 
decision of the content experts from both coxmtries to adopt the text of the competency 
statements as a basis for the examination blxieprint. 
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Phase 2 involved determining whether assigning the same weights to the competency 
statements would be appropriate for assessing entry-level pharmacy practice in both the 
United States and Canada. A survey instrument was developed to allow pharmacy 
practitioners in both countries to assess the seriousness of the consequences associated with 
the competency and subcompetency statements, and how frequently an entry-level 
practitioner would likely be required to perform the activity, or address the issue. Those 
ratings of criticahty and frequency were then combined to obtain overall estimates of the 
relative importance of the competency and subcompetency statements. The estimates of 
importance were then transformed into percentages of the total examination, with the more 
important statements being represented by larger percentages and less important statements 
represented by smaller percentages. The percentage of the examination to any 

particular statement was proportional to its estimated importance. 

Working independently, the American and Canadian panels of content experts both 
reached the same conclusion regarding the weights that should be adopted in establishing 
the blueprint. Each panel independently recommended adopting the blueprint that was 
based on the "bottom up" analysis, where the ratings were based on both American and 
Canadian respondents, and where criticality was weighted four times as heavily as frequency 
in estimating the importance of each competency and subcompetency statement. With the 
bottom up analysis, the importance of each of the thirty-nine subcompetency statements was 
determined and transformed into a percentage of the total examination. The of 

the examination associated with each of the nine competency statements was then 
by su mmin g the percentages associated with the subcompetency statements that comprised 
the competency statement. 

Limitatio ns of the Study 

Tlie most serious limitation relates to the low overall response rates, and particularly 
the response rate for Canadian practitioners. Given the steps that were taken to encourage 
participation, a higher response rate had been expected. A second limitation was the fact 
that only geographic representation was available as a stratifying variable in selecting the 
sample. It would have been preferable to also stratify on the basis of number of years in 
practice, and amount of time spent with entry-level pharmacists. This information was 
collected in the demographic section, but it was only available to describe the sample, not 
to select the sample initially. It would also have been desirable to conduct further analyses 
to assess whether there was any impact from having both French and En gtigh surveys 
available in Canada. Some preliminary analyses showed little difference between the French 
and English responses, but the French sample size was too small to serve as a basis for 
drawing reliable conclusions. 
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Conclusions 



Overall, the procedures reported here to establish and validate a joint examination 
blueprint for use in two coimtries worked very well. Based on comments from simvey 
respondents, and comments from the various reviewers and panel members, some of the key 
factors that contributed to the success of the procedures were the following: 

1. Having panels from both coimtries review the results independently and reach their 
own conclusions before coming together as a group was very helpful. It enabled 
them to review the results objectively, without trying to accommodate the other 
panel, and it made it possible to ^tablish an appropriate blueprint for one coimtry 
or the other, regardless of whether both countries could agree on a single blueprint. 

2. Including descriptors on the points associated with the five-point rating scales reduced 
the chances that respondents would apply different interpretations to the numerical 
values. 

3. Conducting both the bottom up and top down analyses was seen as valuable even 
though the top down approach did not produce workable results. Given the time, 
effort, and expense that go into conducting a practice analysis study, it is important 
if at aU possible to produce results that are workable. In general it seems likely that 
the bottom up analyses will yield more satisfactory results in studies such as these, 
in part because they are based on many more data points (78 in this case, versus 18 
for the top down analysis), and in part because the subcompetency statements tend 
to be more specific, and therefore more amenable to ratings of frequency anH 
criticality. Some reviewers indicated that the competency statements tended to be 
quite broad, makin g it more difficult to estimate their frequency and criticality. 
Because they were so broad, they didn’t tend to be performed in isolation, but rather 
as part of other activities. 

4. Steps taken to help improve response rates included: simplifying the survey 

instructions; shortening the survey; sending out advance notice of the survey to 
encourage participation; and sending a follow-up reminder. Even though were 
not completely successful for the current study, at least every effort was being madp 
to avoid the problems associated with low response rates. 
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Appendix A 



Entry-Level Pharmacy Practice Competencies 
Area 1: Manage I >ruy Tlieraov to Opritnire Patient Outcomes 



Competency 


Criticality 

Rating 


Frequency 

Rating 


1.1 Evaluate the patient and/or patient infonnation to 

determine the presence of a disease or medical condition, 
to determine the need for treatment and/or teferml, and 
to identify patient-specific factors that affect health, 
pharmacotherapy, and/or disease management 


1 2 3 4 5 


1 2 3 4 5 


l.Ll Identify and/or use instruments and techniques related 
to patient assessment and diagnosis 


1 2 3 4 5 


1 2 3 4 5 


l.Ll Identify and define the terminology, signs, and 
symptoms associated with diseases and medical 
conditions 


1 2 3 4 5 


1 2 3 4 5 


1.1.3 Identify drug and non-drug methods of preventing and 
treating diseases and medical conditions 


1 2 3 4 5 


1 2 3 4 5 


1.1.4 Identify patient factors, biosodal factors, and 

concurrent drug therapy that are relevant to the 
maintenance of wellness and the prevention or 
treatment of a disease or medical condition 


1 2 3 4 5 


1 2 3 4 5 



1.2 Assure the appropriateness of the patient’s specific 

pharmaoothcrapcotic agents* icginiens* dosage 

forms* routes of administration* and delivery systems 


Criticality 

Rating 


Frequency 

Rating 


12..1 Identify drug products by their generic, trade, and/or 
common names 


1 2 3 4 5 


1 2 3 4 5 


1.2.2 Identify the known or postulated sites and nv^chanigma 
of action of phannacotherapeutic agents 


1 2 3 4 5 


1 2 3 4 5 


1.2.3 Evaluate drug therapy for the presence of 

phannacotherapeutic duplications and interactions 


1 2 3 4 5 


1 2 3 4 5 


IJ.A Identify indications, contraindications, warnings, and 
precautions associated with a drug product’s active and 
inactive ingredients 


1 2 3 4 5 


1 2 3 4 5 



er|c 



BEST 



1.2.5 Identify physicochemical properties of drug substances 
that affect their solubility, pharmacoldnetics, 
pharmacologic actions, and stability 


1 2 3 4 5 


1 2 3 4 5 


1.2.6 Interpret and apply pharmacokinetic principles to 
calculate and determine appropriate drug dosing 
regimens 


1 2 3 4 5 


1 2 3 4 5 


1.2.7 Interpret and apply biopharmaoeutic principles, and the 
pharmaceutical characteristics of drug dosage forms and 
delivery systems, to assure bioavaiJability and enhance 
patient compliance 


1 2 3 4 5 


1 2 3 4 5 



1.3 Monitor the patient and/or patient information and 

manage the drug regimen to promote health and asaore 
safe and efifective pharmacotherapy 


Criticality 

Rating 


Frequency 

Rating 


1.3.1 Identify pharmacotherapeutic outcomes and endpoints 


1 2 3 4 5 


1 2 3 4 5 


1.3.2 Evaluate patient information to determine the safety 
and effectiveness of pharmacotherapy 


1 2 3 4 5 


1 2 3 4 5 


1.3.3 Identify, describe the mechanism of, and remedy 
adverse reactions and iatrogenic or drug-induced 


1 2 3 4 5 


1 2 3 4 5 


1.3.4 Prevent, recognize, and remedy noncompliance and 
drug misuse or abuse 


1 2 3 4 5 


1 2 3 4 5 


1.3.5 Identify and remedy interactions or contraindications 
with diagnostic or monitoring tests or procedures 


1 2 3 4 5 


1 2 3 4 5 



Area 2: Assure the Safe and Accur a te Preparation and Dis p ensiny r.f M erfi'catinry; 



Competency 


Criticality 

Rating 


Frequency 

Rating 


2.1 Pcrfoim calculations required to compound, dispense, 

and administer medications 


1 2 3 4 5 


1 2 3 4 5 


2.1.1 Calculate the quantity of medication to be compounded 
or dispensed; reduce and enlarge formulation quantities 
and calculate the quantity of ingredients needed to 
compound the proper amount of the preparation 


1 2 3 4 5 


1 2 3 4 5 


2.1.2 Calculate nutritional needs and the caloric content of 
nutrient sources 


1 2 3 4 5 


1 2 3 4 5 


2.1.3 Calculate the rate of drug fldmmistrfttion 


1 2 3 4 5 


1 2 3 4 5 


2.1.4 Calculate or convert drug concentrations, ratio 
strengths, and/or extent of ionization 


1 2 3 4 5 


1 2 3 4 5 



2.2 Select and dispense medications 


Criticality 

Rating 


Frequency 

Rating 


2.2.1 Determine whether a particular drug dosage strength or 
dosage form is commercially available, and whether it is 
available on a nonprescription basis 


1 2 3 4 5 


1 2 3 4 5 


2.2.2 Identify commercially available drug products by their 
characteristic physical attributes 


1 2 3 4 5 


1 2 3 4 5 


2.2.3 Identify the rationale for including exdpients in the 

formulation of a commercial drug product, and predict 
the effects and adverse effects of these formulation 
factors 


1 2 3 4 5 


1 2 3 4 5 


2.2.4 Interpret and apply pharmacokinetic parameters and 
quality assurance data to determine equivalence among 
manufactured drug products, and identify products for 
which documented evidence of inequivalence exists 


1 2 3 4 5 


1 2 3 4 5 


2J2.5 Identify the appropriate packaging, storage, handling 
and disposal of medications 


1 2 3 4 5 


1 2 3 4 5 


2.2.6 Identify and describe the use of equipment and 
apparatus required to administer medications 


1 2 3 4 5 


1 2 3 4 5 
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2.3 Prepare and compoond extemporaneous preparations and 

sterile products 


Criticality 

Rating 


Frequency 

Rating 


2.3.1 Identify and describe techniques and procedures related 
to drug preparation, compounding, and quality 
assurance 


1 2 3 4 5 


1 2 3 4 5 


2.3.2 Identify and use equipment necessary to prepare and 
extemporaneously compound medications 


1 2 3 4 5 


1 2 3 4 5 


2.3.3 Identify the important physicochemical properties of a 
preparation’s active and inactive ingredients; describe 
the mechanism of, and the characteristic evidence of 
incompatibility or degradation; and identify methods for 
achieving stabilization of the preparation 


1 2 3 4 5 


1 2 3 4 5 



Area 3: Provide Prut? Informati on and Promote Public Health 



Competency 


Criticality 

Rating 


Frequency 

Rating 


3.1 Access, evaluate, and apply infonnatioii to promote 

optimal health care 


1 2 3 4 5 


1 2 3 4 5 


3.1.1 Identify the typical content and organization of specific 
sources of drug and health information 


1 2 3 4 5 


1 2 3 4 5 


3.1.2 Interpret and evaluate data presented in textual, 
tabular, or graphic form 


1 2 3 4 5 


1 2 3 4 5 


3.1.3 Evaluate the suitability, accuracy, and reliability of 
infonnation from reference sources by explaining and 
evaluating the adequacy of experimental design and by 
applying and evaluating statistical tests and parameters 


1 2 3 4 5 


1 2 3 4 5 
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3.2 Educate patients and health caxe professionals regarding 

prescription medications, nonpiescription medications, 
and medical devices 


Criticality 

Rating 


Frequency 

Rating 


3.2.1 Provide information regarding a medication's 

therapeutic actions, and describe appropriate remedies 
to minimize the principal untoward effects resulting 
from drug therapy 


1 2 3 4 5 


1 2 3 4 5 


3.2.2 Provide information regarding a medication’s 
precautions, warnings, contraindications, and 
interactions with jfood 


1 2 3 4 5 


1 2 3 4 5 


3.2.3 Provide information regarding the proper storage, 
administration, and disposal of medications 


1 2 3 4 5 


1 2 3 4 5 


3.2.4 Identify products and describe techniques for the self- 
monitoring of patients’ health status 


1 2 3 4 5 


1 2 3 4 5 


3.2.5 Provide advice regarding the selection, use, and care of 
medical/surgical appliances or devices, durable medical 
equipment, and medication administration equipment 


1 2 3 4 5 


1 2 3 4 5 



3.3 Educate patients and the public regarding wellncsa, 

disease states, and medical conditions 


Criticality 

Rating 


Frequency 

Rating 


3.3.1 Provide information regarding medications used in the 
prevention and treatment of diseases and medical 
conditions, including emergency patient care 


1 2 3 4 5 


1 2 3 4 5 


3.3.2 Provide information regarding nutrition, lifestyle, and 
other non-drug measures that are effective in 
promoting health or preventing or minimizing the 
progress of a disease or medical condition 


1 2 3 4 5 


1 2 3 4 5 
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